feat(runtime):增加多模态机制 by phantom5099 · Pull Request #322 · 1024XEngineer/neo-code

phantom5099 · 2026-04-16T11:23:01Z

背景

本 PR 的核心目标是收敛多模态消息在各层的渲染与传递语义，减少重复实现，并清理历史兼容路径导致的语义漂移。

实际改动范围（按模块）

新增共享渲染层：internal/partsrender
统一复用渲染实现：internal/context*、internal/memo、internal/tools、internal/tui/core/app
provider 多模态结构收敛：
- internal/provider/types 新增/扩展内容 part 的结构与转换能力
- internal/provider/openaicompat/chatcompletions 调整请求构造与测试预期
runtime 与会话链路联动：internal/runtime、internal/session 若干路径改为按结构化 parts 流转
相关装配与配置小幅对齐：internal/app、internal/config

关键行为变更

将 Parts -> 文本 的主要规则集中到共享实现，按场景提供明确入口，减少跨模块重复逻辑。
清理图片附件“拼接文本降级”路径，优先保留结构化多模态语义，避免附件语义被文本污染。
provider 侧兼容相关测试断言与当前稳定策略对齐，降低“实现与测试互相拉扯”的风险。

测试与回归

本 PR 同步调整并补充了 context / memo / provider / runtime / session / tui 多处测试。
覆盖重点包括：渲染一致性、parts 结构转换、请求构造、历史消息处理与运行时联动路径。

说明

本次变更不追求引入新的跨层能力，重点是把既有多模态主链路收敛为单一语义来源，并清理过渡期兼容行为。

…sion flow Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

fix(provider): validate parts and simplify message conversion

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

fix(runtime): align todo integration test with parts input

…/todo integration Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

merge: resolve conflicts for PR 1024XEngineer#315

codecov · 2026-04-16T11:24:32Z

Codecov Report

❌ Patch coverage is 90.90909% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/tui/core/app/update.go	73.52%	9 Missing ⚠️
internal/runtime/run.go	81.48%	3 Missing and 2 partials ⚠️
...l/provider/openaicompat/chatcompletions/request.go	91.30%	3 Missing and 1 partial ⚠️
internal/context/parts_render.go	87.50%	1 Missing and 1 partial ⚠️
internal/partsrender/render.go	95.45%	1 Missing and 1 partial ⚠️
internal/provider/types/content_part.go	96.49%	1 Missing and 1 partial ⚠️
internal/runtime/compact_generator.go	83.33%	2 Missing ⚠️
internal/context/internalcompact/messages.go	90.90%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

phantom5099 · 2026-04-16T11:39:17Z

/simplify

fennoai · 2026-04-16T11:39:24Z

已完成你在 /simplify 评论里的要求：检查未解决评审意见、修复值得修复的问题，并执行代码简化复核。

处理结果

已修复评审意见：ValidateParts 增加 ImagePart.SourceType 显式白名单校验，仅接受 remote / session_asset。
已修复评审意见：Message.IsEmpty() 现在把仅包含空白文本 part 的消息视为空，避免空 assistant 消息落库。
已补测试：

internal/provider/types/content_part_test.go 新增不支持图片来源类型用例。
internal/provider/types/message_test.go 新增 IsEmpty 语义覆盖（空白文本、非空文本、图片、tool calls）。

已按要求调用 code-simplifier 子代理，并采纳其简化建议：

ValidateParts 改为更清晰的 switch 结构。
IsEmpty 判定逻辑进一步收敛。

已调整受影响断言：internal/provider/openaicompat/chatcompletions/provider_test.go。

验证

已执行并通过：go test ./internal/provider/... ./internal/runtime/...

提交与推送

Commit: 0232b2c
已推送分支：fork-pr-322-1776338584
fork 同步 PR: fix(provider): tighten multimodal part validation and empty message semantics phantom5099/neo-code#22

未完成项

未执行全仓 go test ./...（本次仅针对受影响 provider/runtime 相关包做验证）。

View job run

…emantics Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

phantom5099 · 2026-04-16T11:46:42Z

@xgopilot 评估一下是否需要增加internal/partsrender，说明具体原因、优劣之处

fix(provider): tighten multimodal part validation and empty message semantics

phantom5099 · 2026-04-16T11:51:40Z

结论

建议保留 internal/partsrender（当前阶段“需要”）。

核心原因：这次改动后，Parts -> 文本 的同一语义已经被 context、memo、tools、tui 多处消费，如果不抽共享层，语义漂移会很快再次出现。

具体依据

共享规则已形成“单一语义源”

internal/partsrender/render.go:17-30 统一了 3 个入口：RenderCompactPromptParts / RenderTranscriptParts / RenderDisplayParts。

internal/partsrender/render.go:32-81 把图片占位策略集中，避免每层各自拼接 [Image] 规则。

跨模块复用是实需，不是提前抽象

目前直接依赖该包的有：

internal/context/parts_render.go:10-23

internal/context/compact/parts_render.go:8-10

internal/memo/parts_projection.go:10-13

internal/tools/format.go:134-136

internal/tui/core/app/parts_render.go:8-10

这说明它已经覆盖 runtime 主链路中的多个读取场景，而不是单点调用。

清理了历史“图片转文本拼接”降级路径

旧的 composeMessageWithImageAttachments 已移除（internal/tui/core/app/input_features.go）。

发送路径改为结构化 Parts（internal/tui/core/app/update.go:430-435），并在资产存储未就绪时显式阻断（413-418），避免继续污染文本语义。

优劣分析

优点

降低语义漂移：图片占位、remote/session_asset 展示规则不再分散。

降低安全风险：display 模式统一不泄露原始图片细节（render.go:58-73）。

便于测试：internal/partsrender/render_test.go 可独立验证核心渲染规则。

缺点

增加一层间接调用：各模块新增薄封装函数，阅读链路会多跳一次。

模式增长风险：若后续出现大量“场景特化渲染”，partsrender 可能膨胀成大杂烩，需要及时拆分。

当前仍有重复判断：hasRenderableParts / hasMemoRelevantUserInput 这类“是否有语义输入”逻辑尚未统一，后续仍可能漂移（虽不阻塞本次结论）。

不建议的替代方案

如果移除 internal/partsrender，把渲染逻辑回填到 context/memo/tui/tools 各层，会回到“多处维护同一规则”的状态；结合本 PR 多模块联动范围，这个回退方案风险更高。

本次完成情况

已完成：仅做架构评估与取舍分析（无代码改动）。

未执行：未实现任何重构或行为调整（本次触发评论仅要求评估）。

View job run

@xgopilot 那么请你具体说明为什么不是放在utils层，以及通用工具函数和多层共享函数之间的具体差别在哪

phantom5099 · 2026-04-16T11:53:55Z

结论

不是放 utils，核心原因是：这里不是“无业务语义的通用小函数”，而是“跨层共享的领域语义策略（ContentPart -> 文本投影）”。

internal/partsrender/render.go 实际承载了三种场景语义（compact prompt / transcript / display），并内置了图片脱敏与来源分支策略；这属于领域策略层，不是纯工具层。

为什么不放到 utils 层

utils 的边界应是无状态、无领域语义、可任意复用的原子能力（如字符串切分、时间格式化）。

partsrender 依赖 provider/types.ContentPart，并理解 ImageSourceRemote / ImageSourceSessionAsset（internal/partsrender/render.go:48-80），已经和消息协议强耦合，不是“通用函数”。

该逻辑包含安全与产品语义：display 模式统一输出 [Image] 防止展示层泄露细节（internal/partsrender/render.go:58-73）。这种策略不应伪装成工具函数。

仓库当前也没有全局 internal/utils 约定；已有 utils 是 TUI 内聚辅助（internal/tui/core/utils），作用域是 UI 子域，不承担跨域语义。

“通用工具函数” vs “多层共享函数”

通用工具函数（utils）

输入输出是基础类型（string/int/time 等）

不依赖业务模型

不表达策略选择

放错位置的代价主要是可读性

多层共享函数（shared domain function）

输入输出是领域类型（这里是 []ContentPart）

显式承载规则与策略分支（3 种 render 模式）

直接影响一致性与安全语义

放错位置会导致语义漂移（各层复制一份规则）

在本 PR 的具体落点

context、memo、tools、tui 都只做薄调用，不再各自维护图片占位规则：

internal/context/parts_render.go:11-24

internal/memo/parts_projection.go:11-13

internal/tools/format.go:134-136

internal/tui/core/app/parts_render.go:8-10

这就是“多层共享函数”的典型形态：单一语义源 + 多处消费，但不是“无语义工具箱”。

可执行判定标准（以后可复用）

若函数依赖领域类型并表达策略分支，优先独立成 internal/<domain> 包，不进 utils。

若函数只处理基础类型且不含业务决策，可放 utils（且限定作用域，例如 tui/core/utils）。

若跨 3 个以上模块复用且语义必须一致，优先建“共享语义层”而非复制粘贴。

本次完成情况

已完成：架构解释与边界判定（无代码改动）。未执行：实现调整与测试（本次仅答疑）。

View job run

@xgopilot 据此，请你分析当前整个项目中有什么适合抽到utils层的工具函数

pionxe · 2026-04-16T12:28:28Z

/review

phantom5099 · 2026-04-16T12:59:48Z

/simplify

fennoai · 2026-04-16T12:59:55Z

Completed

Implemented worthwhile fixes for unresolved review comments in this PR branch and pushed the result.

What I changed

Fixed transcript persistence privacy risk by redacting image identifiers in transcript rendering:
- internal/partsrender/render.go
- Transcript mode now emits [Image:remote] / [Image:session_asset] without URL/query or asset ID.
Tightened memo extraction trigger to avoid low-signal image-only turns:
- internal/memo/parts_projection.go
- hasMemoRelevantUserInput now requires non-empty text parts.
Added/updated tests to lock behavior:
- internal/context/compact/transcript_store_test.go
- internal/partsrender/render_test.go
- internal/memo/parts_projection_test.go
- internal/memo/llm_extractor_test.go

Validation

Passed: go test ./internal/partsrender ./internal/memo ./internal/context/compact

Code simplifier agent

Attempted twice as requested, but both sub-agent runs failed with upstream 502 Bad Gateway.
I still applied local simplification in touched logic (reduced branching in memo input check and kept render branching minimal while preserving behavior).

Git

Commit: 26fc9d4
Pushed branch: fork-pr-322-1776338584
Fork PR (created by push workflow): fix(multimodal): redact transcript image identifiers and tighten memo gating phantom5099/neo-code#24

Not done

Did not run full repository test suite (go test ./...), only targeted affected packages.

View job run

… gating Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

fix(multimodal): redact transcript image identifiers and tighten memo gating

phantom5099 · 2026-04-16T13:07:08Z

/review

phantom5099 · 2026-04-16T13:14:10Z

/simplify

fennoai · 2026-04-16T13:14:17Z

完成情况

已按你这条 /simplify 指令执行：读取历史评论并处理未解决评审意见、评估并修复值得修复项、补测试、并尝试调用 code-simplifier 子代理。

已修复

修复输入丢失问题：当图片发送被拦截时不再提前清空输入。

文件：internal/tui/core/app/update.go
调整：把常规 input.Reset() 下移到附件校验通过之后，避免失败分支直接 return 导致文本丢失。

修复 remote 图片 URL 空白校验缺失。

文件：internal/provider/types/content_part.go
调整：ValidateParts 对 ImageSourceRemote 使用 strings.TrimSpace(image.URL) == "" 判空。

测试补充

internal/tui/core/app/update_test.go
- TestUpdateSendWithUnsupportedImageInput 增加“输入保留”断言。
- 新增 TestUpdateSendWithImageAttachmentsWithoutSessionAssets，覆盖 session asset 未就绪分支并验证输入保留。
internal/provider/types/content_part_test.go
- 新增 remote image blank url 用例。

验证

已通过：

go test ./internal/tui/core/app ./internal/provider/types

code-simplifier 子代理

已按要求尝试两次调用，但两次都因上游 502 Bad Gateway 失败；因此本次由主代理直接完成最小化简化与收敛（未引入额外结构重构）。

提交与推送

Commit: a154b10
已推送分支：fork-pr-322-1776345239
fork PR: fix(multimodal): preserve blocked input and validate blank image url phantom5099/neo-code#25

未执行

未运行全仓测试 go test ./...（仅运行受影响包）。

View job run

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

fix(multimodal): preserve blocked input and validate blank image url

phantom5099 · 2026-04-16T13:20:37Z

/review

fennoai

Focused review completed. I found 2 issues that are still worth addressing.

fennoai · 2026-04-16T13:26:16Z

 		}

-		content := strings.TrimSpace(message.Content)
+		content := strings.TrimSpace(renderCompactPromptParts(message.Parts))


renderCompactPromptParts(message.Parts) currently keeps raw remote URL and session-asset identifier text (via partsrender compact mode). This compact prompt is sent to the model, so signed URL query tokens / internal asset IDs can be re-exposed in later compaction calls. Consider using a redacted projection here (for example transcript/display-safe placeholders) instead of raw identifiers.

fennoai · 2026-04-16T13:26:16Z

+					return errors.New("remote image part must contain url")
+				}
+			case ImageSourceSessionAsset:
+				if image.Asset == nil || image.Asset.ID == "" {


session_asset validation checks image.Asset.ID == "" but does not trim whitespace. IDs like " " will pass validation. Please align with remote URL validation by using strings.TrimSpace(image.Asset.ID) == "" to avoid accepting invalid payloads.

phantom5099 · 2026-04-16T13:36:10Z

/code 修复编译错误，提高测试覆盖率

fennoai · 2026-04-16T13:36:18Z

完成情况

已按你的要求完成“修复编译错误 + 提高测试覆盖率”，并推送到当前工作分支。

本次改动

修复编译错误：更新 internal/tui/core/app 测试里过期的 Message.Content 字段用法，统一改为结构化 Parts。
- internal/tui/core/app/update_test.go
- internal/tui/core/app/view_test.go
提高覆盖率：新增/补充多模态相关分支测试。
- 新增 internal/provider/streaming/accumulator_test.go，覆盖累积器 nil 接收者、工具调用聚合顺序、参数增量拼接、错误分支。
- 补充 internal/provider/types/message_test.go，覆盖 Message.Validate() 正常/异常路径。
- 补充 internal/partsrender/render_test.go，覆盖图片占位渲染的回退分支（空 URL、空 asset、缺失 id、无 mime 等）。

验证结果

go build ./... 通过
go test ./... 通过
重点包覆盖率验证：
- go test ./internal/provider/streaming -cover => 59.5%（由 0% 提升）
- go test ./internal/partsrender -cover => 97.1%
- go test ./internal/provider/types -cover => 93.8%

提交与推送

Commit: 5b7a8c0
已推送：fork-pr-322-1776345239
Fork PR（同步创建/更新）：test: fix compile errors and improve multimodal test coverage phantom5099/neo-code#26

未完成项

未直接修改上游 PR phantom5099:main -> 1024XEngineer:main 的 head 分支（本次按工作流推送到自动化工作分支并同步 fork PR）。

View job run

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

test: fix compile errors and improve multimodal test coverage

phantom5099 and others added 10 commits April 16, 2026 01:05

feat(runtime\provider):初步重构多模态机制

64e6bc1

fix(tui): remove legacy compatibility path and restore multimodal/ses…

0728dc6

…sion flow Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

fix(provider): validate parts and simplify message conversion

6618ee6

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #19 from phantom5099/fork-pr-315-1776306016

9bdd63a

fix(provider): validate parts and simplify message conversion

fix(runtime): align todo integration test with parts input

7107222

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #20 from phantom5099/fork-pr-315-1776306016

c300874

fix(runtime): align todo integration test with parts input

merge(main): resolve PR 1024XEngineer#315 conflicts for runtime parts…

e874d44

…/todo integration Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #21 from phantom5099/fork-pr-315-1776306016

13eec8c

merge: resolve conflicts for PR 1024XEngineer#315

Merge remote-tracking branch 'upstream/main'

8472088

pref(runtime\provider):重构多模态机制

491af12

This comment was marked as outdated.

Sign in to view

phantom5099 mentioned this pull request Apr 16, 2026

pref(runtime)：优化多模态渲染 #321

Closed

fix(provider): tighten multimodal part validation and empty message s…

0232b2c

…emantics Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #22 from phantom5099/fork-pr-322-1776338584

f917dbc

fix(provider): tighten multimodal part validation and empty message semantics

This comment was marked as outdated.

Sign in to view

1024XEngineer deleted a comment from fennoai Bot Apr 16, 2026

xgopilot and others added 2 commits April 16, 2026 13:03

fix(multimodal): redact transcript image identifiers and tighten memo…

26fc9d4

… gating Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #24 from phantom5099/fork-pr-322-1776338584

738ed47

fix(multimodal): redact transcript image identifiers and tighten memo gating

This comment was marked as outdated.

Sign in to view

xgopilot and others added 2 commits April 16, 2026 13:18

fix(multimodal): preserve blocked input and validate blank image url

a154b10

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #25 from phantom5099/fork-pr-322-1776345239

88414d5

fix(multimodal): preserve blocked input and validate blank image url

fennoai Bot reviewed Apr 16, 2026

View reviewed changes

Merge branch 'main' into main

d27da6a

1024XEngineer deleted a comment from fennoai Bot Apr 16, 2026

修复错误

e3271a9

xgopilot and others added 2 commits April 16, 2026 13:42

test(pr322): fix tui compile break and increase multimodal coverage

5b7a8c0

Generated with [codeagent](https://github.com/qbox/codeagent) Co-authored-by: phantom5099 <245659304+phantom5099@users.noreply.github.com>

Merge pull request #26 from phantom5099/fork-pr-322-1776345239

8ac26e4

test: fix compile errors and improve multimodal test coverage

phantom5099 merged commit 3cb534c into 1024XEngineer:main Apr 16, 2026
2 checks passed

fennoai Bot mentioned this pull request Apr 16, 2026

风险点计算 #313

Closed

Uh oh!

Conversation

phantom5099 commented Apr 16, 2026 • edited by fennoai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

背景

实际改动范围（按模块）

关键行为变更

测试与回归

说明

Uh oh!

codecov Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as outdated.

Uh oh!

phantom5099 commented Apr 16, 2026

Uh oh!

fennoai Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

处理结果

验证

提交与推送

未完成项

Uh oh!

phantom5099 commented Apr 16, 2026

Uh oh!

phantom5099 commented Apr 16, 2026

结论

具体依据

优劣分析

不建议的替代方案

本次完成情况

Uh oh!

phantom5099 commented Apr 16, 2026

结论

为什么不放到 utils 层

“通用工具函数” vs “多层共享函数”

在本 PR 的具体落点

可执行判定标准（以后可复用）

本次完成情况

Uh oh!

pionxe commented Apr 16, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

phantom5099 commented Apr 16, 2026

Uh oh!

fennoai Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Completed

What I changed

Validation

Code simplifier agent

Git

Not done

Uh oh!

phantom5099 commented Apr 16, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

phantom5099 commented Apr 16, 2026

Uh oh!

fennoai Bot commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

完成情况

已修复

测试补充

验证

code-simplifier 子代理

提交与推送

未执行

phantom5099 commented Apr 16, 2026 •

edited by fennoai Bot

Loading

codecov Bot commented Apr 16, 2026 •

edited

Loading

fennoai Bot commented Apr 16, 2026 •

edited

Loading

fennoai Bot commented Apr 16, 2026 •

edited

Loading

fennoai Bot commented Apr 16, 2026 •

edited

Loading

fennoai Bot commented Apr 16, 2026 •

edited

Loading